Asymmetric data mirroring

ABSTRACT

An asymmetric data mirroring method with a local storage device and a remote storage device being separated by large distances is disclosed. A server determines a predetermined time period associated with a round trip latency between the server and a remote storage device. The server submits a request to a local storage device and to the remote storage device. The server submits additional requests to the local and remote storage devices during the predetermined time period. The server determines whether an acknowledgement associated with the request has been received from the remote storage device during the predetermined time period. The server resubmits the request and the additional requests to the remote storage device if the acknowledgement is not received. If an acknowledgement is received, the server continues submission of additional new requests to the local and remote storage devices.

FIELD OF THE INVENTION

The present invention relates generally to data mirroring and moreparticularly to a method of mirroring data in a remote location.

BACKGROUND OF THE INVENTION

Data backup is well known. Traditionally, data backup consisted ofcopying data onto a bulk archival media such as a tape on a periodicbasis. Tape backup has been replaced or supplemented more recently bydisk redundancy systems. A disk redundant system typically includes atleast two storage devices such as disk storage devices in whichidentical data is stored. If data in one device is lost or damaged as aresult of the device malfunctioning or becoming disabled, the data isretrieved from the other device. In order to protect data, redundantsystems are sometimes maintained at different geographic locations. Thatis, in order to protect data stored at a location from systemmalfunctions, weather disasters or other types of unforeseen conditions,redundant data is stored at different (or remote) physical locations.Data redundancy is also referred to as data mirroring and typicallyinvolves the submission of simultaneous ‘write’ requests to multiplestorage devices.

In mission-critical applications, maintaining synchronization betweendata at mirrored sites is highly desirable. Synchronization in thiscontext means that all mirrored storage devices must acknowledge receiptof an I/O request from a requesting application before the applicationmay generate the next I/O request. As the distance between the mirroredsites increases, synchronization becomes harder to achieve usingexisting mirroring techniques and methods as the application generatingthe I/O request is slowed while awaiting acknowledgement from the remotestorage device. Typically, in a data mirroring arrangement, a server isattached or connected to a local data storage device as well as to aremote data storage device with the data from each storage devicemirroring that of another. It is possible to obtain synchronizationusing existing techniques if the physical distance between the mirroredsites is less than approximately twenty-five (25) miles. For greaterdistances (i.e., greater than twenty-five miles), existing techniques donot provide synchronicity that is needed for maintaining data securityin case of a wide-spread calamity.

SUMMARY OF THE INVENTION

Methods and systems consistent with the present invention address thislimitation by providing a mechanism through which data may be mirroredin remote storage devices in a near synchronized manner.

A method consistent with the present invention includes a serverdetermining a predetermined time period (or time interval) associatedwith a round trip latency between the server and a remote storagedevice. The server submits a request to a local storage device and tothe remote storage device. The server then submits additional requeststo the local and the remote storage devices during the predeterminedtime period. The server determines whether an acknowledgement associatedwith the request has been received from the remote storage device duringthe predetermined time period. The server resubmits the request and theadditional requests to the remote storage device if an acknowledgementis not received within the time interval. If an acknowledgement isreceived within the time interval, the server continues to submitadditional requests to the local and remote storage devices.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings, which are incorporated in and constitute apart of the specification, illustrate an embodiment of the inventionand, together with the description, explain the invention. In thedrawings,

FIG. 1A illustrates a data mirroring system according to exemplaryembodiments of the present invention;

FIG. 1B illustrates a server according to exemplary embodiments of thepresent invention;

FIG. 1C illustrates a remote storage device according to exemplaryembodiments of the present invention;

FIG. 2 illustrates a data mirroring method according to exemplaryembodiments of the present invention; and

FIG. 3 illustrates a table representing relationship between thefrequency of request generation as a function of roundtrip latency andthe corresponding block latency between the storage devices according toexemplary embodiments of the present invention.

DETAILED DESCRIPTION

The following description of the implementations consistent with thepresent invention refers to the accompanying drawings. The samereference numbers in different drawings identify the same or similarelements. The following detailed description does not limit theinvention. Instead, the scope of the invention is defined by theappended claims.

Methods and systems consistent with the present invention provide amechanism through which effective data mirroring is achieved overprevious distance limitations.

Existing data mirroring methods include synchronous, asynchronous,semi-synchronous and spoofing. Effectiveness of the current synchronousmethod is limited to a twenty-five (25) mile physical separation betweenmirrored sites.

A data mirroring system 100 is illustrated in FIG. 1A. The datamirroring system 100 includes a server 110, a local storage device 120and a remote storage device 130. The server 110 and the remote storagedevice 130 may be connected via a communication link 140. Thecommunication link 140 may be a cable connection or a wirelessconnection. The cable may be terrestrial, underwater, etc. It mayutilize a fiber optic medium. The network formed by the arrangement ofFIG. 1A may be a public or a private network.

There exists an asymmetry in the distance between the server 110 and thestorage devices 120 and 130. The distance between server 110 and localstorage device 120 is negligible relative to the distance between server110 and remote storage device 130. In an asymmetric data mirroring (ADM)method according to exemplary embodiments of the present invention, theserver 110 first submits an I/O request (such as a ‘write’ request of ablock of data) to both the local and remote storage devices 120 and 130and then continues to make additional I/O requests to the devices 120and 130 over a predetermined time period while waiting for anacknowledgement from the remote storage device 130 for the submitted I/Orequest. According to this exemplary embodiment, an acknowledgement(within the predetermined time period or time interval) is required foreach submitted request. The predetermined time period represents thetime needed for a signal to traverse the round trip distance between theserver 110 and the remote storage device 130 plus the maximum ‘write’time of the remote storage device 130. This time period may also bereferred to as the round trip latency or network latency and may bemeasured or determined by a processor, such as processor 112 of FIG. 1B.The ‘write’ time of the remote storage device 130 may be negligiblesince the request may first be written to a cache, such as cache 138 ofFIG. 1C for example, associated with the remote storage device 130.Therefore, an acknowledgement may be submitted by the remote storagedevice 130 upon receipt of the ‘write’ request from the server 110.

It may be appreciated by those skilled in the art that latency betweennetwork nodes, such as server 110 and remote storage device 130, mayvary as a result of network traffic, etc. Accordingly, multiple readingsmay be made in order to determine the round trip latency. That is, anumber of ‘pings’ may be submitted by the server 110 to determine theaverage latency while also noting the minimum and maximum round triplatency values. Furthermore, the latency will change if the distancebetween the server 110 and the remote storage device 130 is changed forany reason. In this case (i.e., the distance between the server and theremote storage device changes), the latency measurements have to beupdated.

The request from the server 110 may be generated by the processor 112 ofFIG. 1B. The round trip latency may also be measured or determined bythe processor 112. A timer, such as timer 16 of FIG. 1B, may beassociated with each submitted block of data for measuring the elapsedtime from the time the block of data (I/O request) is submitted. A copyof each submitted block of data is stored by the server 110 in memory,such as cache memory 118 of FIG. 1B or another type of memory, until anacknowledgement for that block of data is received. For each requestsubmitted, the server waits for an acknowledgement. Upon receiving anacknowledgement, the copy of block of data corresponding to the receivedacknowledgement is erased from the memory 118 of the server 110. Therequests may be submitted and an acknowledgement received by acommunication means 114 of server 110. The remote storage device 130, asillustrated in FIG. 1C, may include a processor 132, a communicationmeans 134 and a cache 138. The communication means 114 and 134 may beimplemented using hardware or software such as a process or a script.

If the server 110 does not receive an acknowledgement from the remotestorage device 130 within the predetermined time period, all furtherrequests to the devices 120 and 130 are halted. At this point, therequest for which an acknowledgement is not received as well as alladditional requests that have been submitted are resubmitted, block byblock, by the server 110 to the remote storage device 130. Theresubmitted blocks of data, as described above, are stored in the server110 such as in cache118 or another type of memory associated with theserver 110 for example. A timer 116 for each submitted block of data isstarted. If the server 110 again fails to receive an acknowledgementwithin the predetermined time period, the server ceases to makeadditional requests and assumes that there is an error. The error may bea read/write error, a transport error or a network error. The server 110then seeks detection and correction of an error either in the transportmedium 140 or in the remote storage device 130.

Due to the possible variation in the round trip latency, as describedabove, a certain tolerance level may be added to the predetermined timeperiod. As a result, the server 110 may factor in this tolerance levelin waiting for an acknowledgement. The tolerance level may be thedifference between the average round trip latency value and the maximumround trip latency value mentioned previously or a percentage of theaverage round trip latency value. In cases of successive or repeatedfailures in receiving acknowledgements from the remote device within thetime interval, the latency values may be re-measured.

FIG. 2 illustrates an exemplary embodiment of the asymmetric datamirroring method of the present invention. A data mirroring process 200commences in step 210. The server 110 measures a predetermined timeperiod T (round trip latency) in step 215. As described, T representsthe time needed for a signal to traverse the round trip distance betweenthe server 110 and the remote storage device 130 plus the maximum‘write’ time of the remote storage device 130. At step 220, the server110 submits an I/O request (such as a ‘write’ function in the form of ablock of data) to the local and the remote storage devices 120 and 130.A timer t measures the time from submission of the request by the server110. The timer t is compared to the round trip latency T at step 225.While t remains less than T, the server 110 checks for the existence ofadditional requests that are to be submitted to the storage devices 120and 130 at step 230. If there are additional requests, the server 110submits I/O requests to the storage devices 120 and 130 at step 220. Ifthere are no additional requests, as determined in step 230, the writingprocess comes to a stop in step 235. If the distance between the server110 and the remote storage device changes, then the process can berestarted at step 210.

When the timer t equals the round trip latency T, the server 110determines whether an acknowledgement from the remote storage device 130has been received at step 240. The server checks for the existence ofadditional requests to be submitted to the storage devices 120 and 130at step 230 and the determination is followed either by an end to thewriting process at step 235 or submission of additional requests to thestorage devices 120 and 130 at step 220 as described above.

If an acknowledgement has not been received, the server 110 resubmits tothe remote storage device 130, at step 245, all requests (block byblock) that have previously been submitted to the storage devices 120and 130 since the last acknowledgement (if any) while halting submissionof new requests. For each request submitted, a timer is started and acopy of the submitted request is maintained by the server 110. The timert again measures time from the submission of the requests and comparesit to the predetermined round trip latency T at step 250. If anacknowledgement from remote storage device 130 is again not received bythe server 110 at step 255, an error state is triggered at step 260. Onthe other hand, if an acknowledgement is received at step 255, theserver checks for additional requests to be submitted to the remotestorage device at step 230 and the process is continued as describedabove.

While the above description indicates an error alert being triggeredafter failure to receive an acknowledgement for a request that has beensubmitted twice, this number may be increased or decreased. That is, theerror state can be triggered after a request has been submitted once andthe server has failed to receive an acknowledgement for this request. Onthe other hand, a particular request may be submitted more than twotimes and a failure to receive an acknowledgement more than two timescan trigger the error alert.

The process may be restarted from 235 upon the generation of a new I/Orequest. In this case, there is no need to re-compute the round triplatency as long as the distance between the server 110 and the remoteserver 130 remains constant. The process may then resume at step 220.Similarly, though not illustrated in FIG. 2, the process may resume atstep 220 after the error at step 260 is detected and corrected.

A timer as described above may be a plurality of timers each associatedwith a submitted request from the server 110. Upon receiving anacknowledgement of the receipt of a particular request, a timerassociated with that request may be reset and associated with a nextsubmitted request. As each block of data is submitted by the server, acopy of the block of data is stored in a cache or another type of memoryassociated with the server until an acknowledgement of the receipt ofthe submitted block of data is received by the server. If anacknowledgement is received, the timer corresponding to the block ofdata for which the acknowledgement is received is reset and the copy ofthis block of data is erased from the server. This facilitatesresubmission of previously submitted requests in case of failure by theserver in receiving an acknowledgement. The functions of submitting ablock of data, ‘writing’ of the submitted block of data by both thelocal and remote storage devices and submitting an acknowledgement bythe remote storage device are known and therefore are not describedfurther.

Due to the physical separation between server 110 (and local storagedevice 120) and the remote storage device 130, there is a differencebetween the data stored in the local device and that stored in theremote device at a particular point in time. That is, a number of blocksof data may be ‘written’ to the local storage device 120 before they are‘written’ in the remote storage device 130 as a result of the differencein distance between the server and each of the storage devices and thetime needed to traverse this distance (i.e., round trip or networklatency). This number of blocks is the maximum number of blocks of datathat may have to be resubmitted by the server to the remote storagedevice upon failure by the server to receive an acknowledgementaccording to exemplary embodiments of the present invention.

The number of blocks may depend on a plurality of factors. One factor isthe round trip latency or the time needed for a signal to traverse theround trip distance between the server and the remote storage device.This amount of time may also be viewed as the time needed to transport ablock of data from the server to the remote storage device plus the timeneeded for an acknowledgement signal to travel from the remote storagedevice to the server. Another factor is the frequency at which the I/Orequests are generated by the server.

The relationship between the round trip latency (previously designatedT) in seconds, the number of blocks (designated delta Δ) and thefrequency of I/O requests (designated lambda λ) in seconds may berepresented by:([T/λ]+1)=Δ.

As the frequency of the generation of I/O requests (as a function of theround trip latency) varies, the number of blocks (of data) by which thelocal and remote storage devices differ also changes. Some sample valuesfor the difference in the number of blocks is illustrated in FIG. 3.

As the frequency of I/O requests increase, the number of blocks by whichthe remote storage device 130 differs from the local storage device 120increases. Knowledge of this number (i.e., difference in blocks) fromoffered load and network latency facilitates construction and managementof a cache in the server or in a network interface device to compensatefor the difference between the contents of the two storage devices.

The asymmetric data mirroring method differs from each of the existingsynchronous, asynchronous, semi-synchronous and spoofing methods. In theexisting synchronous mirroring method, the server submits a writerequest to both the local and remote storage devices and waits for anacknowledgement from the remote device before a next write request issubmitted. That is, until an acknowledgement is received from both thelocal and the remote storage devices, a subsequent request is notsubmitted to either of the storage devices. A predetermined time periodis implemented within an application performing the data mirroring. Thetime period is based on a need to maintain a reasonable user orapplication response time. The remote device must respond (oracknowledge) receipt within this time period or an I/O halt is invoked.Given the need for maintaining a reasonable time period, the distancebetween the local and remote storage devices is limited to a shortdistance. Currently, this method is only effective if the distance islimited to 25 miles (or 40 kilometers).

The ADM method according to the present invention utilizes a differentmethod for writing data to the local and remote storage devices. In ADM,as described, an I/O request is transmitted to both the local and remotedevices. In contrast to existing synchronous methods, however,subsequent I/O requests to the storage devices 120 and 130 are submittedwhile waiting for an acknowledgement from the remote device130. ADMwaits for an acknowledgement from the remote device for a time periodequal to the round trip latency as described above. ADM utilizes theround trip latency time to submit the additional requests and, at aminimum, keeps the contents of the local storage device updated in theinterim time period of waiting for an acknowledgement from the remotestorage device.

In an asynchronous mirroring method, I/O requests are submitted by theserver to the mirrored pair of data storage devices without requiring ablock-by-block acknowledgement. In this method, an amount of data iswritten and a checksum is performed. If an error is detected, therequests are halted. ADM requires the server to receive acknowledgementof previously submitted I/O requests from the remote storage deviceblock-by-block within a predetermined time period.

In the semi-synchronous data mirroring method, the server is programmedto submit a predetermined number of I/O requests without having toreceive acknowledgement of previously submitted I/O requests. When thisnumber of I/O requests is submitted without receiving anacknowledgement, additional I/O requests are halted. ADM requiresacknowledgement of the receipt by the storage device of I/O requests(block by block) in order to continue submission of I/O requests by theserver to the mirrored storage devices. There is no predetermined numberof unacknowledged receipts that are allowed as in the semi-synchronousdata mirroring method. Furthermore, in semi-synchronous method, there isno time element. That is, the I/O requests are halted upon apredetermined number of previously submitted requests not having beenacknowledged irrespective of the amount of time that has elapsed. Asdescribed, in ADM, a pre-determined time period equaling the round triplatency is allocated within which an acknowledgement from the storagedevices is to be received by the server before submission of additionalI/O requests.

In the spoofing mirroring method, applicable in both asynchronous andsemi-synchronous modes, the server prematurely assumes acknowledgementof an I/O request by the storage device; that is, before the storagedevice actually acknowledges receipt of an I/O request. ADM differs fromthe spoofing method as well since an actual acknowledgement,block-by-block, is needed in order to continue submission of I/Orequests to the storage devices.

The foregoing description of exemplary embodiments of the presentinvention provides illustration and description, but it is not intendedto be exhaustive or to limit the invention to the precise formdisclosed. Modifications and variations are possible in light of theabove teachings or may be acquired from practice of the invention.

The following claims and their equivalents define the scope of theinvention.

1. A data mirroring method comprising: determining a predetermined timeperiod associated with a round trip latency between a server and aremote storage device; submitting a request from the server to a localstorage device and to the remote storage device; submitting additionalrequests to the local and remote storage devices during saidpredetermined time period; determining whether an acknowledgementassociated with said request has been received from the remote storagedevice during said predetermined time period; resubmitting said requestand said additional requests to the remote storage device if theacknowledgement is not received; and continuing submission of additionalrequests to the local and remote storage devices if the acknowledgementis received.
 2. The data mirroring method of claim 1, wherein the serverhalts submission of new requests to each of the local and remote storagedevices if the acknowledgement is not received.
 3. The data mirroringmethod of claim 1, wherein a copy of each request submitted by theserver is stored in a memory associated with the server while the serverwaits for the acknowledgement.
 4. The data mirroring method of claim 3,wherein the copy of a request is erased from the memory upon receipt ofan acknowledgement associated with the request.
 5. The data mirroringmethod of claim 1, wherein a timer is started for each request submittedby the server, said timer designed to count up to the determined roundtrip latency.
 6. The data mirroring method of claim 1, wherein thesubmitted requests are resubmitted a predetermined number of times uponfailure to receive an acknowledgement.
 7. The data mirroring method ofclaim 6, wherein a failure in receiving an acknowledgement apredetermined number of times triggers an error alert.
 8. The datamirroring method of claim 1, wherein the requests are in the form ofblocks of data and an acknowledgement is required for each block.
 9. Thedata mirroring method of claim 1, wherein the predetermined time periodis increased by a tolerance level.
 10. The data mirroring method ofclaim 1, wherein the data in the remote storage device differs from thelocal storage device by a number of requests, said number depending onthe roundtrip latency.
 11. The data mirroring method of claim 1, whereinthe data in the remote storage device differs from the local storagedevice by a number of requests, said number depending on a frequency ofgeneration of requests by the server.
 12. The data mirroring method ofclaim 11, wherein an increase in the frequency of request generation bythe server results in an increase in the number of requests by which theremote storage device differs from the local storage device.
 13. A datamirroring system including a server, a local storage device and a remotestorage device, the server comprising: processing means for measuring around trip latency between the server and the remote storage device andfor generating I/O requests; communication means for submitting thegenerated I/O requests to the storage devices and for receivingacknowledgements from the storage devices; memory means for storing acopy of the submitted I/O requests; and monitoring means for measuringelapsed time after submission of an I/O request, wherein thecommunication means: resubmits previously submitted I/O requests to thestorage devices if an acknowledgement is not received when the elapsedtime equals the round trip latency; and continues submission of I/Orequests while the elapsed time is not equal to the round trip latencyor if an acknowledgement is received.
 14. The data mirroring system ofclaim 13, wherein the processing means halts submission of new requestsif the acknowledgement is not received.
 15. The data mirroring system ofclaim 13, wherein the communication means is implemented using acombination of hardware and software.
 16. The data mirroring system ofclaim 13, wherein the monitoring means is at least one timer.
 17. Thedata mirroring system of claim 13, wherein the copy of the I/O requeststored in the memory means is erased upon reception of anacknowledgement corresponding to the I/O request.
 18. The data mirroringsystem of claim 13, wherein the memory means is a cache memory.
 19. Thedata mirroring system of claim 13, wherein the remote storage devicecomprises a communication means and a memory means.
 20. Acomputer-readable medium containing instructions for controlling atleast one processor to perform a data mirroring method comprising:determining a predetermined time period associated with a round triplatency between a server and a remote storage device; submitting arequest from the server to a local storage device and to the remotestorage device; submitting additional requests to the local and remotestorage devices during said predetermined time period; determiningwhether an acknowledgement associated with said request has beenreceived from the remote storage device during said predetermined timeperiod; resubmitting said request and said additional requests to theremote storage device if the acknowledgement is not received; andcontinuing submission of additional requests to the local and remotestorage devices if the acknowledgement is received.